fix(agc): reapply pipeline cache persistence with guarded key queries - #15169

What

I reapply #13931, reverted in #14839: driver pipeline caches survive source updates, and eligible compute pipelines are captured and restored with VK_KHR_pipeline_binary, with ordinary driver-cache fallback. Against merged main d893997f1a I keep main's recorder error handling and debt entries, and link the store test against libSceAgcDriver because DeviceProcLookups moved out of the header. After merging main d893997f1a, the binary layout key also tells main's scratch-pool layouts apart (0, 1 or 2 additional descriptor sets); layouts without a scratch pool keep the same contribution to the key.

New in this PR: global and per-pipeline key queries write into separate, 64-byte-aligned heap scratch followed by a 64-byte guard filled with 0xa5, and a key is accepted only after VK_SUCCESS, an intact guard, the expected sType, a null pNext and a size from 1 to VK_MAX_PIPELINE_BINARY_KEY_SIZE_KHR.

 PipelineBinaryStore(...)
   resolve vkGetPipelineKeyKHR ...
-  getKey(device, nullptr, &globalKey)
-  enable if VK_SUCCESS and 1 <= keySize <= 32
+  getKey(device, nullptr, &guardedScratch.key)
+  enable only if VK_SUCCESS, guard intact, sType unchanged, pNext null, 1 <= keySize <= 32
+  copy the validated key into globalKey
+  else: stay disabled, log once, use the driver pipeline cache

A rejected per-pipeline key falls back to an ordinary build for that pipeline. I have not found the cause of the NVIDIA Linux startup crash in the code: the call uses the Khronos function-pointer type from the device resolver, the output structure is initialised, and the feature is enabled before the call. These checks detect malformed keys and writes that reach the guard; they cannot contain arbitrary driver corruption or clobbered registers, and they cannot stop a crash inside the call. APS5_NO_PIPELINE_BINARIES still turns the store off before the call. I am keeping this PR as a draft so it can be checked on the affected machine first.

Tested

Windows, NVIDIA GeForce RTX 4080, driver 610.88. No title or performance claim.

• main vs this PR, same machine and GPU, ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ctest -j 1 -V, the 134 existing tests registered in the files this PR touches (pipeline cache, recorder and Vulkan test-device tests): main d893997f1a 134/134, this PR c8244e0269 134/134. The validation layer loaded in both logs, neither reports a missing requested layer, and neither has a VUID- or SYNC-HAZARD- line.
• Store tests: 12/12 pass. The new cases cover the query arguments, zero, oversized and maximum key sizes, a changed sType or non-null pNext for global and per-pipeline keys, a corrupted guard and the fallback after a rejected global key. The new tests require the added scratch type and IsReady; their passing results do not reproduce the Linux crash. With the startup test, agc_pipeline_binary_vulkan and three agc_pipeline_cache_allocation modes, all 17 tests this PR adds pass under the settings above.
• agc_driver_pipeline_binary_startup (new): creates the production device with the shader cache enabled in a temporary directory and requires the store to be ready when the device has VK_KHR_pipeline_binary (exit 77 with skipped, the device lacks VK_KHR_pipeline_binary otherwise). Here it prints pipeline binary startup: global-key query completed successfully; store ready; device=NVIDIA GeForce RTX 4080; driver=NVIDIA 610.88.
• Cache enabled (neither ANYPS5_NO_SHADER_CACHE nor APS5_NO_PIPELINE_BINARIES set, validation layer loaded): 83/83 pass: the startup test, the binary-store and pipeline-cache tests, agc_driver_small_copy_sync, both agc_driver_address_passes tests, the other agc_driver tests that do not disable the cache, and the automatically added agc_driver_pm4_replay fixture (which disables the shader cache itself). No VUID- line. agc_pipeline_binary_vulkan saves binary records for two pipelines and restores both in the next process. On main d893997f1a, the 65 of these tests that exist there, plus the fixture, pass 66/66. The binary round-trip test uses trivial shaders and an empty layout, so it does not test scratch-pool layouts.

This machine never reproduced the crash, so these runs show the path works here, not that the Linux crash is gone. @oneandonlydean, you offered to test a candidate on the affected RTX 4090: could you run this on that installation after building this branch?

env -u ANYPS5_NO_SHADER_CACHE -u APS5_NO_PIPELINE_BINARIES \
  ANYPS5_REQUIRE_VULKAN=1 \
  VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -j 1 -V \
  -R '^(agc_driver_pipeline_binary_startup|agc_pipeline_binary_(vulkan|store.*)|agc_pipeline_cache.*|agc_driver_small_copy_sync|agc_driver_address_passes_(targets_split|draw_transitions))$'

The most useful result for me would be the startup line (store ready), the [pipeline-binary] disabled: unusable global key (...) line, which names why the key was rejected and reports the driver-cache fallback, or a backtrace if vkGetPipelineKeyKHR still crashes. A run without the validation layer, as in your report, would also help. I will mark this ready for review only after that result.

Checklist

• Based on current main; no other open PR implements the same functions
  Merged with main d893997f1a, without a force-push; a trial merge with the newer d48f11ab16 has no conflicts. Before this merge, tools/pr_overlap.py reported conflicting lines with fix(agc): build one compute pipeline when results differ only in a table's size #8212, perf(agc): GPU descriptor heap on VK_EXT_descriptor_heap (APS5_GPU_HEAP) #9508, perf(agc): copy only the vertex records a draw fetches #9875, fix(agc): read capture deltas and replay them while queues wait on the title #12567, fix(agc): leave GPU-stored words out of frame capture deltas #12568, fix(agc): relaunch the frame replay when a host region blocks a captured GPU range #12696, perf(recompiler): back compute scratch with a device buffer slot pool #12764, fix(agc): write back BDA stores into writable heap mirrors #12840 and test(agc): enable the portability subset with its features in the recorder tests' device #13942 (ARCHITECTURE.md, VulkanDevice.cpp, Recorder.cpp), and some comparisons could not be completed. A GitHub search found no open PR implementing driver pipeline caches or pipeline binaries.
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  No unimplemented exports or silent stubs are added.
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
  No third-party code is added.
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: NVIDIA GeForce RTX 4080, driver 610.88, Windows
• AI-assisted: yes

🤖 Generated with Claude Code